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Summary:  The  Canadian  Automated  Pilot  Selection 
System  (CAPSS)  is  a computerized  simulator  of  a 
single  engine  light  aircraft  used  in  the  selection  of 
pilots  for  the  Canadian  Forces.  This  paper  describes 
the  characteristics  of  the  CAPSS  simulator  and  the 
types  of  data  it  collects.  The  development  of  the 
CAPSS  equation  that  predicts  the  probability  of 
success  in  flying  training  is  discussed  and  the  results  of 
the  initial  validation  and  cross-validation  are  presented. 
Demographic  characteristics  of  applicants  assessed  by 
CAPSS  since  its  introduction  in  February,  1997,  are 
presented.  Finally,  some  of  the  strengths  and 
weaknesses  of  CAPSS  are  discussed. 

The  assessment  of  officer  applicants  for  tire 
Canadian  Forces  follows  the  same  general  procedure 
for  all  officer  entry  programs.  An  additional  step  in  the 


assessment  of  pilot  applicants  is  testing  with  the 
Canadian  Automated  Pilot  Selection  System  (CAPSS). 
CAPSS  is  a computerized  moving-base  simulator  of  a 
single  engine  light  aircraft.  The  CAPSS  simulator 
presents  the  candidates  with  all  the  information 
necessary  to  perform  flight  manouevres  using 
instrument  flying  procedures.  A computer  system  runs 
the  simulator,  presents  all  instructions  and  feedback, 
and  collects  the  data.  The  entire  syllabus  consists  of 
five  one-hour  sessions;  the  first  four  sessions  are 
currently  used  for  assessing  pilot  applicants.  The 
sessions  proceed  from  basic  instructions  and  flying  a 
simple  straight  and  level  manoeuvre  to  flying  a 
complex  traffic  pattern.  The  CAPPS  computer  system 
stores  each  candidate’s  data  and  produces  summary 
statistics  of  each  candidate’s  performance.  An  outline 
of  the  complete  syllabus  is  presented  in  Table  1. 


Table  1.  CAPSS  Syllabus 


Session 

Tasks 

1 

Basic  flight  instruments  and  controls;  straight  and  level  flight;  straight  climb;  straight 
descent 

2 

Review;  take-off,  climb-out  and  level  off;  level  turns;  standard  rate  turns 

3 

Review;  climbing  and  descending  turns 

4 

Review;  rectangular  course;  airport  traffic  pattern 

5 

Review;  airport  traffic  pattern  with  side  tasks;  landing 

Note:  Session  5 is  not  used  in  actual  pilot  selection 

Ten  flight  parameters  are  monitored  and 
recorded  twice  per  second  while  each  candidate  is 
“flying”,  although  not  all  parameters  are  relevant  to 
every  flight  task.  The  parameters  are  heading,  turn 
rate,  bank  angle,  side  slip,  altitude,  vertical  speed, 
engine  speed,  airspeed,  and  “x”  and  “y”  Cartesian  co- 
ordinates indicating  current  position.  This  results  in 
1200  data  points  per  candidate  for  every  minute  of 
simulator  operation. 

Dr.  Barry  Spinner  of  the  University  of  New 
Brunswick  developed  the  plan  for  taking  the  mass  of 
data  produced  by  CAPSS  and  transforming  it  into  a 
single  equation  used  to  predict  the  probability  of 
success  at  flying  training  (Spinner,  1988a,  1988b, 
1989, 1990).  The  overall  plan  of  analysis  was  to  divide 
the  data  according  to  the  basic  manoeuvre  being 
performed.  Within  each  manoeuvre,  the  data  were 
broken  into  a number  of  discrete,  statistically 
manageable  data  segments.  Within  each  of  theses  data 


segments,  Summary  Measures  (SMs)  were  formed  to 
characterize  candidate  operation  of  the  simulator  over  a 
brief  period  of  time.  Tire  SMs  within  a data  segment 
were  reduced  using  statistical  methods,  then  the 
segments  were  combined  into  subgroups  and  the  SMs 
were  again  reduced.  This  cycle  of  reduction  and 
combination  was  repeated  within  each  manoeuvre  to 
produce  a combination  of  SMs  that  optimally  predicted 
performance  in  flying  training. 

A hierarchy  of  CAPSS  flight  tasks  was 
designed  to  organize  the  data  and  to  specify  the  order 
in  which  these  subsets  were  combined.  Flight  tasks 
were  categorized  according  to  the  type  of  manoeuver 
being  flown,  tire  context  within  which  the  flight  task 
was  performed  (simple  practice  or  part  of  a more 
complex  flight  task),  which  repetition  of  the  task  was 
being  performed,  tire  flight  parameter  being  monitored, 
and  whether  the  data  were  extracted  from  the 
beginning  or  end  of  the  flight  task  (see  Figure  1). 


Paper  presented  at  the  RTO  HFM  Workshop  on  “Officer  Selection", 
held  in  Monterey,  USA,  9-11  November  1999,  and  published  in  RTO  MP-55. 
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Eight  basic  flight  manoeuvres  represent  the  highest  manoeuvre  category  was  formed  for  data  segments 

level  of  the  hierarchy:  straight  and  level  flight,  straight  taking  place  just  before  and  after  the  occurrence  of  a 

climb,  straight  descent,  level  turn,  climbing  turn,  side  task  in  session  5. 

descending  turn,  take-off,  and  landing.  A ninth 

All  Manoeuvres 


Basic  Manoeuvres:  Straight  and  Level  Other  Eight  Manoeuvres 


Complexity: 

1 

Simple 

1 

1 

Rectangular 

1 

1 

Traffic 

1 

Repetitions: 

1 1 1 
1 2 3 

1 

1 1 
1 to  9 

1 1 

1 to  6 

Segments: 

1 1 
1 2 

1 

Parameters: 

1 1 
1 2 

1 1 

3 4 

1 

5 

Raw  Data:  10  to  60 

Instrument  Readings 

Figure  1.  Hierarchical  Structure  for  Straight  and  Level  Flight,  from  Spinner,  1990 


Summary  Measures 

The  following  description  of  the  development 
and  intended  meaning  of  the  SMs  is  taken  from 
Spinner,  1989  and  1990. 

“The  formation  of  SMs  had  the  goal  of 
transforming  the  raw  data  into  meaningful  variables 
that  represent  the  candidates’  operation  of  the  simulator 
over  a specified  period  of  time.  The  raw  data  that  are 
provided  by  CAPSS  consist  of  instrument  readings 
(e.g.,  compass  headings)  taken  twice  per  second.  In 
this  form,  the  data  represent  the  state  of  the  simulator, 
but  do  not  indicate  the  candidate’s  proficiency  in 
operating  it.  It  is  only  when  these  readings  are 
transformed  into  SMs  that  meaningful  information 
about  the  candidate’s  ability  may  be  obtained.  It  is 
these  SMs  that  were  used  as  predictors  in  analyzing  the 
candidates’  performance  in  flying  training  (Spinner, 
1990,  pp.  6-7).” 

“Many  of  these  SMs  rely  upon  reference  to 
the  command  level  (CL),  which  is  the  target  value  for 
the  parameter  (e.g.,  the  required  heading  or  altitude). 
Other  SMs  refer  to  warnings  which  occur  as  tire 
candidate  deviates  more  and  more  form  the  CL: 
anticipatoiy  warnings  (first  level),  out-of-tolerance 
warnings  (second  level),  and  crash  preventions  which 
cause  the  simulator  to  freeze  and  reset  (Spinner,  1989, 

p.  12).” 

“The  SMs  were  devised  specifically  for  this 
project  based  upon  previous  empirical  literature,  the 


expert  opinions  of  CF  flight  instructors  and  flight  crew, 
and  the  author’s  insights  gained  form  operation  of 
CAPSS  and  flying  experience.  The  SMs  were 
designed  to  assess: 

a.  accuracy  in  keeping  flight 
instruments  on  target; 

b.  variability  in  performance; 

c.  speed  in  response  to  errors  and 
warnings; 

d.  smoothness  of  operation  and 
avoidance  of  overcorrections  and 
abrupt  movements; 

e.  coordination  of  the  flight  controls, 
and 

f.  number  of  instances,  and  amount  of 
time  spent  flying  towards  or  away 
from  die  command  level...  (Spinner, 
1990,  p.  7)” 

The  main  study  which  lead  to  the  present 
CAPSS  equation  was  the  prediction  of  candidate  flying 
training  performance  in  Primary  Flight  Training  (PFT) 
and  in  Basic  Flight  Training  (BFT)  or  Rotaiy  Wing 
Training  (RWT;  Spinner,  1990).  Spinner  used  the 
flying  training  results  of  a sample  of  309  male 
anglophone  pilot  candidates  who  had  already  been 
selected  for  pilot  training  by  the  aircrew  selection  tests 
then  in  place.  For  this  sample,  62. 1%  passed  PFT,  and 
of  those,  76.6%  passed  BFT  or  RWT.  The  criterion 
used  in  the  study  was  success  or  failure  in  flying 
training.  Each  candidate  was  assigned  to  one  of  four 
groups: 
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a.  candidates  who  failed  PFT; 

b.  candidate  who  passed  PFT  and  failed 
BFT; 

c.  candidates  who  passed  both  PFT  and 
BFT,  and; 

d.  candidates  who  passed  both  PFT  and 
RWT  (none  of  the  candidates  in  the 
sample  failed  RWT). 

Equations  predicting  flying  training  outcome 
were  developed  for  each  of  the  five  CAPSS  sessions 
through  discriminant  analysis  using  a four  group 
criterion  (fail  PFT,  fail  BFT,  pass  BFT,  pass  RWT). 
Each  session’s  equations  were  based  on  three 
discriminant  functions.  The  best  discrimination  was 
between  those  who  failed  PFT  and  those  who  passed 
PFT,  and  most  of  this  information  was  derived  from 
Hour  1.  Hours  2 through  5 added  relatively  more 
discrimination  between  those  who  failed  BFT  and 
those  who  passed  advanced  flying  training  (BFT  or 
RWT). 

These  equations  were  used  to  generate  each 
candidate’s  predicted  probability  of  passing  flying 
training.  The  classification  results  for  each  equation 
were  then  computed.  Using  Hour  5 results  showed 
CAPSS  to  have  an  overall  correct  prediction  rate  of 
81.9%.  The  percentage  of  candidates  selected  by 


Table  2.  CAPSS  predictive  accuracy 


Pass  Rate1 

Loss  Rate2 

Hour  1 

84.2%  (64/76) 

27.3%  (24/88) 

Hour  2 

88.7%  (63/71) 

28.4%  (25/88) 

Hour  3 

87.5%  (63/72) 

28.4%  (25/88) 

Hour  4 

89.2%  (66/74) 

25.0%  (22/88) 

Hour  5 

85.5%  (65/76) 

26.1%  (23/88) 

Previous  System: 
Current  Sample 

80.0%  (88/1 10) 

unknown 

Previous  System: 
Historical  Rate 

55% 

57% 

Notes: 

1.  The  pass  rate  is  the  number  of  candidates  who  were 
correctly  predicted  to  pass  flying  training,  relative  to 
the  total  number  who  were  predicted  to  pass  (i.e.  of 
those  predicted  to  pass,  the  number  who  actually  did 
pass). 


CAPSS  that  actually  passed  at  PFT  and  BFT/RWT 
were  89.9%  and  83.9%  respectively.  The  loss  rate  (the 
percentage  of  candidates  who  passed  flying  training 
that  were  rejected  by  CAPSS)  was  8.2%.  This  result 
was  compared  to  the  system  that  was  actually  used  to 
select  the  candidates:  a pass  rate  of  62.1%  at  PFT  and 
76.6%  at  BFT.  However,  the  author  notes  that  these 
results  are  inflated  due  to  the  use  of  stepwise 
procedures  in  developing  the  equation  and  applying 
them  to  the  sample  that  generated  them.  A final 
analysis  was  to  examine  the  effect  of  basing  selection 
decisions  on  fewer  than  five  sessions.  The 
classification  results  indicate  that  gains  in  prediction 
accuracy  were  made  until  the  fourth  session,  and  that 
the  fifth  session  added  little  to  prediction  with  this 
sample.  In  practice,  only  four  sessions  are  used  in  pilot 
selection,  and  the  fifth  session  is  not  used. 

Spinner  (1996)  performed  a cross-validation 
of  CAPSS  using  a sample  of  110  male  anglophone 
pilot  candidates.  The  best  discrimination  between 
those  who  passed  flying  training  and  those  who  failed 
occurred  with  Hour  4 results.  The  classification 
decisions  compared  to  actual  results  are  reported  in 
Table  2.  The  author  notes  that  the  CAPSS  pass 
predictions  are  still  higher  than  the  unusually  low 
failure  rate  for  this  sample  in  comparison  to  the 
historical  rate. 


Overall  Correct 
Predictions3 
67.3%  (74/1 10) 
70.0%  (77/1 10) 
69.1%  (76/1 10) 
72.7%  (80/1 10) 
69.1%  (76/1 10) 

unknown 

unknown 


2.  The  loss  rate  is  the  number  of  candidates  who 
actually  passed  flying  training  but  who  were  predicted 
to  fail. 

3.  The  overall  percent  correct  is  the  number  of 
candidates  correctly  predicted  to  either  pass  or  fail, 
relative  to  the  total  number  of  candidates. 
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Table  3.  CAPSS  scores  and  gender  - all  candidates 


Female 

Male 

Session 

Mean 

SD 

N 

Session 

Mean 

SD 

N 

Hour  1 

0.543 

0.219 

69 

Hour  1 

0.649 

0.228 

524 

Hour  2 

0.513 

0.271 

63 

Hour  2 

0.612 

0.250 

492 

Hour  3 

0.466 

0.241 

20 

Hour  3 

0.604 

0.287 

175 

Hour  4 

0.494 

0.348 

20 

Hour  4 

0.527 

0.308 

175 

Pelchat  (1999)  reported  the  demographic 
characteristics  of  a sample  of  756  candidates  assessed 
using  CAPSS  since  its  introduction  as  the  pilot 
selection  in  February,  1997.  During  the 
implementation  of  CAPSS,  candidates  were  only  tested 
to  Hour  2;  in  the  sample  reported  by  Pelchat,  only  207 
candidates  completed  all  four  sessions.  No  significant 
differences  were  found  between  the  performance  of 
Anlgophone  and  Francophone  candidates,  however, 
gender  differences  in  performance  were  apparent. 
Table  3 reports  the  mean  scores  obtained  by  males  and 
females  for  each  of  the  sessions.  CAPSS  scores  are 
reported  as  a probability  of  passing  flying  training;  the 
cutoff  is  a score  of  0.7  in 


Hour  4.  While  males  performed  better  overall,  male 
applicants  were  significantly  more  likely  to  have 
previous  flying  experience  (PFE)  than  female 
candidates.  Only  24%  of  the  females  had  PFE, 
compared  with  42%  of  the  males,  while  32%  of  the 
males  had  over  50  hours  and  almost  20%  had  over  100 
hours,  compared  to  nine  and  three  percent, 
respectively,  for  tire  females.  The  results  of  males  and 
females  with  no  PFE  (Table  4)  are  similar.  As  of  1 
December,  1998,  there  were  only  29  candidates  who 
had  attempted  PFT  and  just  five  who  had  attempted 
BFT.  These  candidates  were  processed  with  two 
CAPSS  sessions,  using  a CAPSS  cut  off  of  0.5.  For 
PFT,  27  candidates  passed,  one  failed  flying,  and  one 
was  recoursed,  for  an  overall  93%  success  rate. 


Table  4.  CAPSS  scores  and  gender  - no  PFE 


Female 

Male 

Session 

Mean 

SD 

N 

Session 

Mean 

SD 

N 

Hour  1 

0.540 

0.207 

44 

Hour  1 

0.547 

0.232 

233 

Hour  2 

0.494 

0.279 

38 

Hour  2 

0.494 

0.246 

222 

Hour  3 

7 

Hour  3 

0.441 

0.242 

57 

Hour  4 

7 

Hour  4 

0.385 

0.258 

57 

Strengths  and  Weaknesses 

The  main  strength  of  CAPSS  is  that  it  is  a 
work  sample  of  the  complex  skills  needed  to  fly  an 
aircraft.  In  selecting  pilots,  a range  of  important  skills 
can  be  identified  and  measured  by  a variety  of  tests;  tire 
advantage  of  CAPSS  is  that  it  measures  these  skills 
simultaneously,  just  as  the  skills  would  be  used  in 
actual  flying. 

The  main  weakness  of  CAPSS  is  that  it  is  not 
an  easy  system  to  modify.  The  CAPSS  predictive 
equation  was  derived  based  on  a sample  of  pilot 
candidates  following  a specific  training  programme, 
using  specific  aircraft.  The  effects  of  changing  the 
training  or  the  introduction  of  new  training  aircraft 
cannot  be  readily  determined.  Insofar  as  CAPSS 
measures  the  “right  stuff”  required  to  become  a pilot,  it 
should  be  relatively  immune  to  these  changes.  In  the 
CF,  validation  of  CAPSS,  or  any  pilot  selection  system, 
is  difficult  because  of  the  small  number  of  pilots  tliat 
are  trained.  This  effect  is  doubled  for  CAPSS  because 
it  uses  performance  results  to  derive  the  equations. 
With  other  systems,  it  may  be  possible  to  simply  tweak 
cutoff  scores  within  a battery  of  tests;  the  development 


of  CAPSS  equations  is  a statistically  complex 
procedure. 
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